Augmenting video games with add-ons

ABSTRACT

A computing system is configured to execute a computer program on a server and to provide a video stream of the program output to a geographically remote client over a communication network. An add-on manager is provided to facilitate the use of add-ons to extend the functionality of the computer program. The add-on manager is responsive to commands received from the client and is configured to associate individual add-ons and add-on data with specific user accounts. The add-ons can be located on the server or some other location remote from the client.

CLAIM OF PRIORITY

This application is a continuation of and claims priority to and thebenefit of U.S. patent application Ser. No. 16/809,489, filed on Mar. 4,2020, entitled “Add-on Management Methods”; which is a continuation ofand claims priority to and the benefit of U.S. patent application Ser.No. 15/883,694, filed on Jan. 30, 2018, entitled “Add-on ManagementMethods”; which is a continuation of and claims priority to and thebenefit of U.S. patent application Ser. No. 13/231,883, filed on Sep.13, 2011, entitled “Add-on Management Methods”; which claims priority toand the benefit of U.S. Provisional App. Ser. No. 61/382,470 filed onSep. 13, 2010, entitled “Add-on Management,” all which are incorporatedherein by reference in their entireties.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to U.S. patent application Ser. No.12/826,130 filed Jun. 29, 2010 and entitled “Video Game Overlay,” andU.S. patent application Ser. No. 12/826,489 filed Jun. 29, 2010 andentitled “Voice Overlay.” The disclosures of all the above commonlyowned patent applications are hereby incorporated herein by reference.

TECHNICAL FIELD

The invention is in the field of computer systems and specifically inthe field of server side execution of computer programs.

BACKGROUND

Add-ons are sets of computing instructions that are used to extend thefunctionality of or otherwise augment a computer program. Add-ons areused in a wide variety of applications. For example, an Adobe Reader®add-on may be used to enable Microsoft word to print to a .pdf file, orto enable a browser to view .pdf files. Add-ons are popular in on-linevideo games. For example, the popular multi-player on-line game World ofWarcraft® has available hundreds of add-ons that can be used to enhancegame play. These add-ons are typically provided by third parties. To addan add-on to a computer program, a user downloads the add-on from awebsite and installs the add-on in an appropriate directory on theuser's computer. As used herein, the term “add-on” is used to refer tooptional computing instructions configured to extend the functionalityof a computer program, where the add-on is developed specifically forthe computer program. An add-on may be developed by a third-party or bythe developer of the computer program. Some computer programs areconfigured to look for add-ons in specific directories. Add-ons aredistinguished from other software utilities, such as printer drivers, inthat they are configured to augment a specific program. Add-ons aresometimes referred to as “plug-ins.”

Add-on data is data included with, generated or organized by an add-on.For example, in a video game add-on data may include the locations ofobjects within a game, inventory information, records of purchases andsales, auction house data, characteristics of non-player characters,maps, historically information, audio data, event timing information,and/or the like.

Computer programs often include computing instructions or an applicationprogramming interface (API) configured for communication between thecomputer program and an add-on. For example, it is common for a computerprogram to be configured to look in a specific directory for add-ons.This directory can be a subdirectory of a directory in which thecomputer program is stored.

SUMMARY

Some embodiments of the invention include systems and methods of usingadd-ons in systems where a computer program is executed on a server andstreaming video generated as a result of the program's execution isprovided to a remote client. The add-ons can be installed on the serveror the client. If an add-on is installed on the client and the computerprogram on a server, then an application programming interface (API) isprovided such that the computer program can operate as if the add-on wason the server local to the computer program. The API on the server isconfigured to pass communications between the add-on and the computerprogram over a communication network.

If the add-on is installed on the server, the computer program may beinstalled on either the client or a location geographically remote fromthe client. Various embodiments of the invention can include, forexample, utilities configured for remotely installing add-ons, computinginstructions configured to provide a virtual environment for executionof the add-on; storage of add-on data, turning add-ons on and off,executing add-ons securely, maintaining an approved add-on list; and/orthe like.

Various embodiments of the invention include game server systemcomprising a video source configured to use game logic to generate videoframes and to encode the video frames to a video stream, the videostream being based on a game environment of a video game, the game logicbeing augmented by an add-on; an I/O device configured to communicatethe video stream to a geographically remote client and to receive gamecommands from the client; add-on storage configured to store the add-onin a location accessible to the video game, and to store add-on data ofthe add-on in a location accessible to the add-on; an add-on managerconfigured to create an add-on environment configured for use of theadd-on by the video game, associate the add-on with a user account, andassociate the add-on data of the add-on with the user account; and aprocessor configured to execute the game logic or the add-on manager.

Various embodiments of the invention include a system comprising a videosource configured to use game logic to generate video frames and toencode the video frames to a video stream, the video stream being basedon a state of a video game and game commands received from ageographically remote client, the game logic being augmented by anadd-on; an I/O device configured to communicate the video stream to thegeographically remote client and to receive the game commands from theclient; an add-on manager configured to automatically update the add-onin response to receiving a new version of the game logic; and aprocessor configured to execute the game logic or the add-on manager.

Various embodiments of the invention include a system comprising gamelogic installed on a first computing device and configured to generatevideo frames and to present the video frames on a display; a first APIfilter installed on the first computing device and configured toredirect commands between the game logic and an add-on, the add-on beinginstalled on a second computing device geographically remote from thefirst computing device and being configured to augment the functionalityof the game logic; an I/O device configured to communicate the commandsbetween the first computing device and the second computing device; anda processor configured to execute the game logic.

Various embodiments of the invention include method of executing anadd-on, the method comprising installing the add-on on a game serversystem configured to provide a video stream to a client, the videostream being a result of execution of game logic whose functionality theadd-on is configured to extend, the client being geographically remotefrom the game server system; executing the game logic; receiving a gamecommand from the client at the game logic; passing the received gamecommand from the game logic to the add-on; generating add-on data usingthe add-on; and storing the add-on data in a location geographicallyremote from the client, the add-on data being associated with an accountof a user of the client.

Various embodiments of the invention include method of installing anadd-on, the method comprising identifying the add-on to be installed;establish an add-on environment configured for the add-on to be accessedby a computer program, the computer program being executed using aclient and being located on a server geographically remote from theclient; placing the add-on in the add-on environment; associating theadd-on with an account of a user of the client; identifying storage foradd-on data generated by the add-on; and associating the storage or theadd-on data with the account.

Various embodiments of the invention include method of automaticallyupdating an add-on, the method comprising receiving an updated versionof a game logic at a video source, the video source being configured togenerate a video stream based on game commands received from ageographically remote client and to provide the video stream to theremote client; determining that an add-on is used to augmentfunctionality of the game logic; sending a request for an updatedversion of the add-on to a third party, the request including anidentity of the add-on and an address to which the update should beprovided; receiving the updated version of the add-on via a computingnetwork, the updated version of the add-on being configured to augmentthe updated version of the game logic; and automatically installing theadd-on in an add-on environment configured for the add-on to be used bythe game logic.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system configured to execute a computer programaugmented by one or more add-ons, according to various embodiments ofthe invention.

FIG. 2 is a block diagram of embodiments of a video source configured toserve multiple video games augmented by add-ons, according to variousembodiments of the invention.

FIG. 3 illustrates further details of an add-on manager, according tovarious embodiments of the invention.

FIG. 4 illustrates further details of a client, according to variousembodiments of the invention.

FIG. 5 illustrates methods of installing and/or using an add-on,according to various embodiments of the invention.

FIG. 6 illustrates methods of automatically updating an add-on,according to various embodiments of the invention.

DETAILED DESCRIPTION

Systems in which a computer program is executed on a server and a videostream output of the computer program is provided to a geographicallyremote client are disclosed in the commonly owned patent applicationscited above. Various embodiments of the invention provide an ability touse add-ons in these systems. The add-ons are installed, executed,supported and otherwise managed by an add-on manager that can be locatedat the server, the client or some other location. For example, theadd-on manager may be used to install an add-on within an appropriateenvironment of the server such that it can be accessed through an API ofthe computer program. Add-on data generated by the add-on and the add-onitself are typically associated with a specific, client, user account,avatar, and/or the like. In other embodiments of the invention thecomputer program but not the add-on is installed on the client.

FIG. 1 illustrates a Game System 100 configured to execute a computerprogram augmented by one or more add-ons, according to variousembodiments of the invention. Game System 100 is configured to provide avideo stream to one or more Clients 110 via a Network 115. Game System100 typically includes a Video Server System 120 and an optional GameServer 125. Video Server System 120 is configured to provide the videostream to the one or more Clients 110 with a minimal quality of service.For example, Video Server System 120 may receive a game command thatchanges the state of, or a point of view within, a video game, andprovide Clients 110 with an updated video stream reflecting this changein state with minimal latency. The Video Server System 120 may beconfigured to provide the video stream in a wide variety of alternativevideo formats, including formats yet to be defined. Further, the videostream may include video frames configured for presentation to a user ata wide variety of frame rates. Typical frame rates are 30 frames persecond, 60 frames per second, and 120 frames per second, although higheror lower frame rates are included in alternative embodiments of theinvention. In some embodiments, Video Server System 120 is configured toprovide three dimensional video data. E.g. matched video streamsrendered from points of view separated by the distance between aperson's eyes.

Clients 110, referred to herein individually as 110A, 110B, etc., mayinclude terminals, personal computers, game consoles, tablet computers,telephones, televisions, set top boxes, kiosks, wireless devices,digital pads, stand-alone devices, handheld game playing devices, and/orthe like. Typically, Clients 110 are configured to receive encoded videostreams, decode the video streams, and present the resulting video to auser, e.g., a player of a game. The processes of receiving encoded videostreams and/or decoding the video streams optionally includes storingindividual video frames in a receive buffer of the client. The videostreams may be presented to the user on a display integral to Client 110or on a separate device such as a monitor or television. Clients 110 areoptionally configured to support more than one game player. For example,a game console may be configured to support two, three, four or moresimultaneous players. Each of these players may receive a separate videostream, or a single video stream may include regions of a framegenerated specifically for each player, e.g., generated based on eachplayer's point of view. Clients 110 are optionally geographicallydispersed. The number of clients included in Game System 100 may varywidely from one or two to thousands, tens of thousands, or more. As usedherein, the term “game player” is used to refer to a person that plays agame and the term “game playing device” is used to refer to a deviceused to play a game.

Clients 110 are configured to receive video streams via Network 115.Network 115 may be any type of communication network between computingdevices including, a telephone network, the Internet, wireless networks,powerline networks, local area networks, wide area networks, privatenetworks, and/or the like. Network 115 explicitly does not includecommunication channels completely within a computing device, such as amotherboard bus. In typical embodiments, the video streams arecommunicated via standard protocols, such as TCP/IP or UDP/IP.Alternatively, the video streams are communicated via proprietarystandards.

A typical example of Clients 110 is a personal computer comprising aprocessor, non-volatile memory, a display, decoding logic, networkcommunication capabilities, and input devices. The decoding logic mayinclude hardware, firmware, and/or software stored on a computerreadable medium. Systems for decoding (and encoding) video streams arewell known in the art and vary depending on the particular encodingscheme used.

Clients 110 may, but are not required to, further include systemsconfigured for modifying received video. For example, a client may beconfigured to perform further rendering, to overlay one video image onanother video image, to crop a video image, and/or the like. Clients 110may be configured to receive various types of video frames, such asI-frames, P-frames and B-frames, and to process these frames into imagesfor display to a user. In some embodiments, a member of Clients 110 isconfigured to perform further rendering, shading, conversion to 3-D, orlike operations on the video stream. A member of Clients 110 isoptionally configured to receive more than one audio or video stream.Input devices of Clients 110 may include, for example, a keyboard, ajoystick, a pointing device, a force feedback device, a motion and/orlocation sensing device, a mouse, a touch screen, a neural interface, acamera, input devices yet to be developed, and/or the like.

The video stream (and optionally audio stream) received by Clients 110is generated and provided by Video Server System 120. As is describedfurther elsewhere herein, this video stream includes video frames (andthe audio stream includes audio frames). The video frames are configured(e.g., they include pixel information in an appropriate data structure)to contribute meaningfully to the images displayed to the user. Ameaningful contribution is a contribution that is readily observable bya game player. As used herein, the term “video frames” is used to referto frames including predominantly information that is configured tocontribute to, e.g. to effect, the images shown to the user. Most of theteachings herein with regard to “video frames” can also be applied to“audio frames.” Clients 110 may be configured to receive more than onevideo stream at the same time. For example, Client 110B may beconfigured to receive a matched pair of video streams configured to forma three dimensional image when one of the streams is presented to oneeye and the other stream is presented to the other eye.

Clients 110 are typically configured to receive inputs from a user.These inputs may include game commands configured to change the state ofthe video game or otherwise affect game play. The game commands can bereceived using input devices and/or may be automatically generated bycomputing instructions executing on Clients 110. The received gamecommands are communicated from Clients 110 via Network 115 to VideoServer System 120 and/or Game Server 125. For example, in someembodiments, the game commands are communicated to Game Server 125 viaVideo Server System 120. In some embodiments, separate copies of thegame commands are communicated from Clients 110 to Game Server 125 andVideo Server System 120. The communication of game commands isoptionally dependent on the identity of the command. Game commands areoptionally communicated from Client 110A through a different route orcommunication channel that that used to provide audio or video streamsto Client 110A.

In various embodiments, Clients 110 are configured to communicate add-ondata between each of Client 100 and Video Server System 120, or betweendifferent members of Clients 110. For example, an add-on that enablesvoice communication directly between clients can include thecommunication of audio data between clients. Add-on data that augmentsthe functionality of a server side computer program may be stored on oneof Clients 110 for use by a server side add-on or for use by a clientside add-on. Add-on data may also be stored on Video Server System 120or on Add-on Server 165, and be used by add-ons located on one ofClients 110, Video Server System 120 or Add-on Server 165.

Game Server 125 is optionally operated by a different entity than VideoServer System 120. For example, Game Server 125 may be operated by thepublisher of a multiplayer game. In this example, Video Server System120 is optionally viewed as a client by Game Server 125 and optionallyconfigured to appear from the point of view of Game Server 125 to be aprior art client executing a prior art game engine. Communicationbetween Video Server System 120 and Game Server 125 can occur viaNetwork 115. As such, Game Server 125 can be a prior art multiplayergame server that sends game state information to multiple clients, oneof which is Video Server System 120. Video Server System 120 may beconfigured to communicate with multiple instances of Game Server 125 atthe same time. For example, Video Server System 120 can be configured toprovide a plurality of different video games to different users. Each ofthese different video games may be supported by a different Game Server125 and/or published by different entities. In some embodiments, severalgeographically distributed instances of Video Server System 120 areconfigured to provide game video to a plurality of different users. Eachof these instances of Video Server System 120 may be in communicationwith the same instance of Game Server 125. Communication between VideoServer System 120 and one or more Game Server 125 optionally occurs viaa dedicated communication channel. For example, Video Server System 120may be connected to Game Server 125 via a high bandwidth channel that isdedicated to communication between these two systems.

Video Server System 120 comprises at least a Video Source 130, an I/ODevice 145, a Processor 150, and non-transitory Storage 155. VideoServer System 120 may consist of one computing device or be distributedamong a plurality of computing devices. These computing devices areoptionally connected via a communications system such as a local areanetwork.

Video Source 130 is configured to provide a video stream, e.g.,streaming video or a series of video frames that form a moving picture.In some embodiments Video Source 130 is also configured to provide anaudio stream. In some embodiments, Video Source 130 includes a videogame engine and rendering logic. The video game engine is configured toreceive game commands from a player and to maintain a copy of the stateof the video game based on the received commands. This game stateincludes the position of objects in a game environment, as well astypically a point of view. The game state may also include properties,images, colors and/or textures of objects. The game state is typicallymaintained based on game rules, as well as game commands such as move,turn, attack, set focus to, interact, use, and/or the like. Part of thegame engine is optionally disposed within Game Server 125. Game Server125 may maintain a copy of the state of the game based on game commandsreceived from multiple players using geographically disperse clients. Inthese cases, the game state is provided by Game Server 125 to VideoSource 130, wherein a copy of the game state is stored and rendering isperformed. Game Server 125 may receive game commands directly fromClients 110 via Network 115, and/or may receive game commands via VideoServer System 120.

Video Source 130 typically includes rendering logic, e.g., hardware,firmware, and/or software stored on a computer readable medium such asStorage 155. This rendering logic is configured to create video framesof the video stream based on the game state. All or part of therendering logic is optionally disposed within a graphics processing unit(GPU). Rendering logic typically includes processing stages configuredfor determining the three-dimensional spatial relationships betweenobjects and/or for applying appropriate textures, etc., based on thegame state and viewpoint. The rendering logic produces raw video that isthen usually encoded prior to communication to Clients 110. For example,the raw video may be encoded according to an Adobe Flash® standard,.wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x. Xvid.FFmpeg, x264, VP6-8, realvideo, mp3, or the like. The encoding processproduces a video stream that is optionally packaged for delivery to adecoder on a remote device. The video stream is characterized by a framesize and a frame rate. Typical frame sizes include 800.times.600,1280.times.720 (e.g., 720p), 1024.times.768, although any other framesizes may be used. The frame rate is the number of video frames persecond. A video stream may include different types of video frames. Forexample, the H.264 standard includes a “P” frame and an “I” frame.I-frames include information to refresh all macro blocks/pixels on adisplay device, while P-frames include information to refresh a subsetthereof. P-frames are typically smaller in data size than are I-frames.As used herein the term “frame size” is meant to refer to a number ofpixels within a frame. The term “frame data size” is used to refer to anumber of bytes required to store the frame.

In alternative embodiments Video Source 130 includes a video recordingdevice such as a camera. This camera may be used to generate delayed orlive video that can be included in the video stream of a computer game.The resulting video stream, optionally includes both rendered images andimages recorded using a still or video camera. Video Source 130 may alsoinclude storage devices configured to store previously recorded video tobe included in a video stream. Video Source 130 may also include motionor positioning sensing devices configured to detect motion or positionof an object, e.g., person, and logic configured to determine a gamestate or produce video-based on the detected motion and/or position.

Video Source 130 is optionally configured to provide overlays configuredto be placed on other video. For example, these overlays may include acommand interface, login instructions, video frames rendered fromanother game player's point of view, messages to a game player, imagesof other game players, video feeds of other game players (e.g., webcamvideo). In embodiments of Client 110A that include a touch screeninterface, the overlay may include a virtual keyboard, joystick, touchpad, and/or the like. In one example of an overlay a player's voice isoverlaid on an audio stream. Video Source 130 optionally furtherincludes one or more audio sources.

In embodiments wherein Video Server System 120 is configured to maintainthe game state based on input from more than one player, each player mayhave a different point of view comprising a position and direction ofview. Video Source 130 is optionally configured to provide a separatevideo stream for each player based on their point of view. Further,Video Source 130 may be configured to provide a different frame size,frame data size, and/or encoding to each of Client 110. Video Source 130is optionally configured to provide 3-D video.

I/O Device 145 is configured for Video Server System 120 to send and/orreceive information such as video, commands, add-ons, add-on data,requests for information, a game state, client identities, playeridentities, game commands, security information, audio data, and/or thelike. I/O Device 145 typically includes communication hardware such as anetwork card or modem. I/O Device 145 is configured to communicate withGame Server 125, Network 115, and/or Clients 110. I/O Device 145 isconfigured to receive the information from more than one of Clients 110.I/O Device 145 is optionally configured to receive the information aspackets using a standard such as TCP or UDP.

Processor 150 is configured to execute logic, e.g. software, includedwithin the various components of Video Server System 120 discussedherein. For example, Processor 150 may be programmed with softwareinstructions in order to perform the functions of Video Source 130, GameServer 125, and/or an Add-on Manager 160. Video Server System 120optionally includes more than one instance of Processor 150. Processor150 may also be programmed with software instructions in order toexecute commands received by Video Server System 120, or to coordinatethe operation of the various elements of Game System 100 discussedherein. Processor 150 may include one or more hardware device. Processor150 is an electronic processor.

Storage 155 includes non-transitory analog and/or digital storagedevices. For example, Storage 155 may include an analog storage deviceconfigured to store video frames. Storage 155 may include a computerreadable digital storage, e.g. a hard drive, an optical drive, or solidstate storage. Storage 115 is configured (e.g. by way of an appropriatedata structure or file system) to store video frames, artificial frames,a video stream including both video frames and artificial frames, audioframe, an audio stream, add-ons, add-on data, and/or the like. Storage155 is optionally distributed among a plurality of devices. In someembodiments, Storage 155 is configured to store the software componentsof Video Source 130 discussed elsewhere herein. These components may bestored in a format ready to be provisioned when needed.

Video Server System 120 optionally further comprises Add-on Manager 160.Add-on Manager 160 is configured to facilitate the use of add-ons toaugment computer programs on Video Server System 120 and/or Clients 110.For example, in some embodiments, Add-on Manager 160 is configured toaugment a game program within Video Source 130. In some embodiments,Add-on Manager 160 is configured to augment a game program withinClients 110.

Tasks that may be performed by Add-on Manager 160 include, but are notlimited to, installing an add-on, updating an add-on, executing anadd-on, storing add-on data, and communicating add-on data. In someembodiments, Add-on Manager 160 includes computing instructionsconfigured for redirecting communications between an add-on and theassociated computer program. Add-on Manager 160 may redirect messagesfrom the computer program, originally directed toward a specificlocation relative to the computer program, to an alternative location.For example, if the computer program included within Video Source 130 isconfigured to access add-ons and/or add-on data in a specific directoryrelative to the computer program, Add-on Manager 160 may be configuredto redirect that access to a different location within Video ServerSystem 120, to one of Clients 110, or to a separate Add-on Server 165.The specific directory in which the computer program is configured toaccess add-ons may be a subdirectory of a directory in which thecomputer program is executed. The location to which access is redirectedis optionally a virtual environment. Alternatively, if the computerprogram is within Client 110B and is configured to access add-ons withina specific directory of Client 110B, then Add-on manager may beconfigured to redirect this access to a location within Video ServerSystem 120 and/or Add-on Server 165. As such, an add-on on Client 110Bcan be used to augment a computer program on Video Server System 120, orvice versa. Likewise, an add-on on Add-on Server 165 can be used toaugment a computer program on either Clients 110 or Video Server System120. A virtual environment is an environment in which some location orphysical device, such as a storage location or path, is simulated usingsoftware.

Add-on Manager 160 includes hardware, firmware, and/or software storedon a computer readable medium. All or part of Add-on Manager 160 isoptionally disposed on a computing device separate from one or moreother elements of Video Server System 120. For example, all or part ofAdd-on Manager 160 is optionally disposed on Clients 110, Add-on Server165, or at some other location within Game System 100. Further detailsof Add-on Manager 160 are discussed elsewhere herein.

Optional Add-on Server 165 is a system configured for storage and/orexecution of one or more add-ons. Add-on Server 165 may also be used tostore add-on data. Communications between Add-on Server 165 and VideoServer System 120 and/or Clients 110 can occur directly or throughNetwork 115. Add-on Server 165 is optionally managed by the publisher ordeveloper of an add-on. In some embodiments, communications from acomputer program are redirected to a location of Add-on Server 165, froma location proximate to the computer program. For example, access toadd-on data or an add-on executable may be redirected from Client 110Bor Video Server System 120 to Add-on Server 165. Game System 100 mayinclude more than one Video Server System 120 and/or more than oneAdd-on Server 165.

Add-on Server 165 includes hardware, firmware, and/or software stored ona computer readable medium. For example, in some embodiments Add-onServer 165 includes a computing device, digital storage, an electronicprocessor and I/O devices. The digital storage can include add-onexecutable code, add-on data, images, all or part of Add-on Manager 160,computing instructions configured to manage add-on's communication withother elements of Game System 100, and/or the like.

FIG. 2 is a block diagram of embodiments of Video Source 130 configuredto serve multiple video games augmented by add-ons, according to variousembodiments of the invention. The multiple video games can includemultiple instances of the same video game and/or instances of differentvideo games. The video games are optionally multiplayer games. Forexample, a game state of one or more of the video games may bemaintained by one or more instances of Game Server 125 each based oninputs received from multiple clients. The elements of Video Source 130illustrated in FIG. 2 are optionally executed using Processor 150.

The video games are executed using different instances of Game Logic210, individually labeled 210A, 210B, 210C, etc. Game Logic 210 isconfigured to use a game state to determine a game environment that canbe rendered to a video stream configured to be presented to a gameplayer. The game environment is a two or three dimensional virtualenvironment including in-game objects, avatars, locations of objects,their shapes, textures, and spatial relationships there between, and thelike. A game environment can include vertex data, transformation dataand texture data, and/or the like.

The rendering of the game environment is typically based on one or morepoints of view associated with a specific game player. These points ofview can be drastically different, such as an overhead view and aforward view. Or, the points of view can be from positions differing byapproximately the distance between a person's pupils, and configured tocreate a 3D image. Video Source 130 may include more than 1, 2, 3, 5, 7,15, or 31 instances of Game Logic 210. Game Logic 210 is optionally aclient of Game Server 125 and may be configured to communicate with GameServer 125 via Network 115.

Game Logic 210 is configured to receive game commands from one or moreof Clients 110 and to process the received commands according to a setof game rules. These rules cover, for example, how avatars interact withother avatars or in game objects, avatar movement, game instancemanagement, and/or the like. Game Logic 210 is optionally alsoconfigured to generate audio data based on events within the game. Thisaudio data may represent a gunshot, a splash, an engine, voice, flying,rain, music, or any other sound that could occur in a game. For example,an event such as one object hitting another may result in audio datarepresenting a related sound. Game Logic 210 includes hardware,firmware, and/or software stored on a computer readable medium.

Each instance of Game Logic 210 can be disposed on a separate computingdevice or several instances of Game Logic 210 could be disposed on thesame computing device or a single instance of Game Logic 210 could bedisposed across multiple computing devices. Instances of Game Logic 210can dynamically change the number and/or identify of computing devicesused depending on the requirements of the game environment the user iscurrently experiencing. Instances of Game Logic 210 disposed on the samecomputing device are optionally executed within separate virtualmachines or virtual I/O shells. In some embodiments, different instancesof Game Logic 210 are configured to communicate game commands and/orgame state information directly to each other, e.g., without necessarilycommunication through Game Server 125.

Game Logic 210 typically executes on top of an Operating System (OS)215. Operating System 215 may include Windows™, Linux, Unix, Mac OS™,Solaris™, and/or the like. A virtual machine platform may operatebetween Operating System 215 and Game Logics 210. The virtual machineplatform can include commercial systems such as ESX, Hyper-V, and/or thelike. In these embodiments, one or more of Game Logic 210 can beexecuted within a virtual machine. Multiple instances of Game Logic 210may execute on the same instance of Operating System 215. For example,FIG. 2 illustrates Game Logic 210A and Game Logic 210B both executing onthe same Operating System 215. Instances of Game Logic 210 executing onthe same Operation System 215 may, or may not, be configured for playingthe same video game. For example, Game Logic 210A, 210B and 210C may allbe World of Warcraft® clients, or may be clients of World of Warcraft®,Eve® and Call to Arms® respectively.

In some embodiments, the game environments determined by Game Logic 210are passed to an optional Virtual 3D Video Driver 220. Virtual 3D VideoDriver 220 is configured to appear, from the point of view of Game Logic210, as a non-virtual 3D video driver controlling a graphics processingunit. Each instance of Game Logic 210 may be associated with its owninstance of Virtual 3D Video Driver 220, or Virtual 3D Video Driver 220may be shared by two or more instances of Game Logic 210. In someembodiments, game audio determined by each Game Logic 210 is passed toan optional Virtual Audio Driver 285. Game audio includes sound dataconfigured to be presented to a game player as part of an audio stream.The game audio can result from rules of the game, e.g., shooting a gunshould result in a popping sound, or may be received from other sourcessuch as music tracks, members of Clients 110, a microphone, and/or thelike.

In alternative embodiments, Game Logic 210 is configured to include thefunctionality, as discussed herein, of Virtual 3D Video Driver 220and/or Virtual Audio Driver 285. In these embodiments, Virtual 3D VideoDriver 220 and/or Virtual Audio Driver 285 are not required.

Virtual 3D Video Driver 220 is further configured to pass the receivedgame environments to a (non-virtual) 3D Driver 230. Optionally thedelivery of game environments to 3D Driver 230 is coordinated by thevarious instances of Virtual 3D Video Driver 220. For example, deliverycan be coordinated such that 3D Driver 230 receives game environmentsfrom only one or a minimum number of Virtual 3D Video Driver 220 at atime. In typical embodiments, each of Virtual 3D Video Driver 220 isconfigured such that they appear to be a separate process and a separatesource of video data to 3D Driver 230. As such, 3D Driver 230 isconfigured to keep track of which video data results in which videoframes after rendering.

The video data received by 3D Driver 230 are passed to GraphicsProcessing Unit (GPU) 235 for rendering into raw video frames. GraphicsProcessing Unit 235 is optionally used to render more than one videostream in parallel. For example, Graphics Processing Unit 235 maygenerate a first video stream based on video data received from GameLogic 210A, generate a second video stream based on video data receivedfrom Game Logic 210B, generate a third video stream based on video datareceived from Game Logic 210C, etc. in parallel. The parallel productionof video streams includes the generation of these streams at the sametime. However, parallel production of video streams may, but does notnecessarily, include the processing of individual frames at the sametime within Graphics Processing Unit 235. For example, in someembodiments 3D Driver 230 alternatively passes the video data generatedby the various members Game Logic 210 to Graphics Processing Unit 235.Data generated by Game Logic 210A is used to make a video frame, andsubsequently data generated by Game Logic 210B is used to make a videoframe, etc. In this case, the video streams are produced in parallelwhile individual frames are produced in series. Alternatively, more thanone video frame may be generated within Graphics Processing Unit 235 atthe same time. In this case a first part of Graphics Processing Unit 235is used to generate one frame while a second part of Graphics ProcessingUnit 235 is used to generate a second frame, the first and second framesarising from video data produced by different Game Logic 210. In someembodiments, one set of graphical computation cores is used to generatethe first frame while a second set of graphical computation cores isused to generate the second frame, at the same time. The resulting videoframes are passed back to the control of 3D Driver 230.

Virtual 3D Video Drivers 220 are optionally configured to manage thetransfer of raw rendered video frames from 3D Driver 230. For example,Virtual 3D Drivers 220 may be used to coordinate the transfer of videoframes from the control of 3D Driver 230 to a Shared Memory 240.(Following rendering, the video frames are in a memory of GraphicsProcessing Unit 235 or a memory managed by 3D Driver 230. In either casethey are under the control of 3D Driver 230.) As Virtual 3D VideoDrivers 220 manages the communication of video data and frames to andfrom 3D Driver 230, in some embodiments, they are capable of placing thevideo frames within parts of Shared Memory 240 associated with specificVideo Dynamic Encoding Pipelines (DEPs) 245. The Video DEPs 245 areindividually identified as Video DEP 245A, Video DEP 245B, Video DEP245C, etc. In these embodiments, each Video DEP 245 is assigned specificmemory locations within Shared Memory 240, and is configured to retrievevideo data from those locations.

In other embodiments, Virtual 3D Drivers 220 are configured to managethe transfer of video data to Shared Memory 240 based on timing. Inthese embodiments, the transfers managed by each of Virtual 3D Drivers220 are synchronized and the Virtual 3D Drivers 220 notify each VideoDEP 245 when data for them is in Shared Memory 240. Once this data isretrieved by the notified member of Video DEPs 245, data for anothermember of DEPs is transferred from 3D Driver 230 to Shared Memory 240under the control of Virtual 3D Video Drivers 220. In these cases pixelsread back from the local GPU memory to Shared Memory 240 can be based ona schedule. As used herein the terms “in sync” and “synchronized” aremeant to mean that the two events are related in time by a schedule, bya timing signal, by a time delay, or that one event does not start untilthe other event has satisfied a condition, e.g., has completed. Forexample, Virtual 3D Drivers 220 may operate in sync such that a gameenvironment is provided to 3D Driver 230 from a first of Virtual 3DDrivers 220 once Graphics Processing Unit 235 has completed rendering ofa game environment from a second of Virtual 3D Drivers 220. The timingof this synchronization is optionally selected to make optimum use ofGraphics Processing Unit 235.

As a result of the management by Virtual 3D Video Drivers 220, themultiple video streams can be stored in Shared Memory 240 withoutconfusing which frames belong to which video stream. Shared Memory 240is optionally configured to store audio as well as video data. Thisaudio data may be stored and/or managed in ways similar to thatdiscussed with respect to video data.

Virtual Audio Drivers 285 are optionally part of a virtual I/O shellbetween Game Logic 210 and OS 215. They are configured to appear fromthe point of view of Game Logic 210 as an audio driver and to pass anyaudio data they receive from Game Logic 210 to one of Audio DEP 270. Forexample, in some embodiments, the Virtual Audio Driver 285 associatedwith Game Logic 210A is configured to send audio data to Audio DEP 270A,and the Virtual Audio Driver 285 associated with Game Logic 210B isconfigured to send audio data to Audio DEP 270B, etc.

One or more of Game Logic 210 are optionally each associated with anAdd-on Environment 287. Add-on Environments 287 include hardware,firmware and/or software stored on a computer readable medium and areconfigured for the associated Game Logic 210 to access add-ons and/oradd-on data. For example, in some embodiments Add-on Environment 287includes a directory structure configured to store add-ons and/or add-ondata. The specific directory structure can depend on the identity ofGame Logic 210A or 210B. For example, if the Game Logic 210A isconfigured to look for add-ons in a subdirectory called “add-ons” and tolook for add-on data in subdirectories of the add-ons directory, thenthe Add-on Environment 287 can include these subdirectories or virtualrepresentations of these subdirectories. Typically, Add-on Environments287 are configured such that Game Logic 210A can operate as if theadd-on were installed locally to Game Logic 210A, e.g., within the samedirectory structure or in the same local file system as Game Logic 210A.This allows the use of instances of Game Logic 210A and add-ons that areconfigured for installation together on the same computing device, evenwhen Game Logic 210 and the associated add-ons are installed on separatecomputing devices coupled by an embodiment of Network 115 such as theinternet.

In some embodiments Add-on Environments 287 include computinginstructions configured to redirect communications between Game Logic210 and add-ons (and add-on data) stored at one of Clients 110, Add-onServer 165, and/or other locations within Video Server System 120. Forexample, Add-on Environments 287 can include computing instructions tosend and receive messages via an API of Game Logic 210 and to redirectthese messages. Add-on Environments 287 optionally include a filterconfigured to identify which messages from the API should be redirectedand to redirect particular messages to different locations as needed. Asimilar filter can be included local to the Add-on. This filter can alsobe considered part of Add-on Environments 287 and is configured toreceive messages (e.g., data or commands) from the add-on and directthese messages to the associated member of Game Logic 210. For example,when an add-on responds to a request or command from Game Logic 210A theresponse to the request or command is redirected back to Game Logic210A. The redirection that occurs in either direction can includecommunication of data packets via network 115. Add-on Environments 287are typically generated by or include parts of Add-on Manager 160.

Shared Memory 240 includes random access memory (RAM) or a similarmemory configured for efficient reading and writing of video data.Shared Memory 240 is configured to store video data for a plurality ofdifferent Video DEP 245. Video data for different Video DEPs 245 isoptionally stored at the same time in Shared Memory 240. Shared Memory240 may consist of a single hardware device or may include multipledevices.

Video DEPs 245 are dynamically allocated encoding pipelines that areeach configured to encode video data rendered using Graphics ProcessingUnit 235. Each member of Video DEPs 245 is configured to encode to videoformats specified at the time the Video DEP 245 is provisioned. Thisformat specification is typically based on the needs of one of Clients110 and/or the capabilities of the communication path between VideoServer System 120 and the Client 110. Video DEPs 245 are optionallyprovisioned dynamically in response from a request from one of Clients110. For example, when Client 110A connects to Video Server System 120and sends a request for a video stream, Video DEP 245 can be provisionedto include elements, such as an encoder, selected specifically to meetneeds of Client 110A. As is discussed elsewhere herein, a member ofVideo DEPs 245 is optionally configured to use more than one encodingscheme.

Video DEPs 245 each include a Grabber 247 configured to grab video datafrom Shared Memory 240 and transfer this video data to a memory of theVideo DEP 245. The Grabber 247 is optionally under the control of amember of Virtual 3D Video Drivers 220. For example, in variousembodiments, Grabber 247 is controlled by a member of Virtual 3D Drivers220 to grab the video data from a specific location within Shared Memory240 and/or at a specific time in synchronization with the transfer ofvideo data to other members of Video DEPs 245.

Video DEP 245 each optionally include an Overlay Logic 290 configured tooverlay one or more video images on the video data, e.g., frames,generated by Graphics Processing Unit 235. An overlay is a first imageto be placed on a second image, or a sound added to another sound.Overlays may be applied with varying levels of transparency. Forexample, the first image can be opaque and, thus, completely hidecovered parts of the second image, or the first image can be partiallytransparent in which case the second image can still be somewhat visibleunder the first image. An overlay may cover all or part(s) of theunderlying image. For example, an overlay may be configured to covercertain pixels within a larger area. In this case the overlay is mappedto pixels on a display of one of Clients 110 or mapped to pixels withinan image generated by Graphic Processing Unit 235. The mapping can beaccomplished in various ways. For example, a lower left pixel of theoverlay may be assigned to a specific pixel on a video frame generatedby Graphics Processing Unit 235. In some embodiments, Overlay Logic 290is configured to resize an overlay prior to applying the overlay.Overlay Logic 290 is optionally located in other positions within VideoDEP 245A. For example, after Preprocessor 250.

The overlays can include a wide variety of pixel/video images. Forexample, an overlay can include a real-time or static image of a gameplayer received via the internet, a real-time or static image of asporting (or other) event, an image of a input control (e.g., “a,” “b,”“TAB,” “Return,” “Space Bar,” “Function Key,” “Arrow,” and/or other keysor input devices), a map, text, and/or the like. Overlays can varywidely in size and shape. In some cases the overlay is the result ofrendering a 3D game environment. More than one overlay can be placed onan image. These overlays can be overlapping or separate. Overlays areoptionally received in an encoded format and decoded prior to beingapplied.

The presence and content of overlays are optionally responsive toreceived game commands, identity of one or more game players, an accounttype, identity of a game being played, video captured from a prior orreal-time sporting event or real-world game, game rules, and/or thecontent of video generated by Game Logic 210. For example, an overlaymay include a menu resulting from receipt of a game command. An overlaymay include content responsive to a pixel pattern, e.g., image, includedwithin video generated by Game Logic 210. Overlay Logic 290 isoptionally configured to apply multiple overlays to a single video frameand these overlays may be from different sources.

Overlay Logic 290 is optionally configured to apply an overlay after avideo frame has been preprocessed using Preprocessor 250 and/or afterencoding using one of Encoders 255. In some embodiments the overlayincludes an image received at an input of Video Source 130 (e.g., I/Odevice 145) via the internet and Overlay Logic 290 includes videoprocessing logic configured to generate the overlay from the image. Thereceived image is optionally received from an image source such as acamera or a file storage.

Video DEPs 245 each optionally also include a Preprocessor (PRE) 250.Preprocessor 250 is configured to perform a color space conversion suchas RGB to YUV and/or a scaling operation to increase or decrease theresolution of the video frame. Preprocessor 250 is optional inembodiments wherein the output of Graphics Processing Unit 235 is in theYUV color space or some other desired color space. MultiplePreprocessors 250 may be included in a Video DEP 245 configured toproduce multiple video streams having video frames of different sizes.

Video DEPs 245 each include at least one Encoder (ENC) 255. Encoders 255are individually identified as Encoder 255A, Encoder 255B, etc. Each ofEncoders 225 is configured to encode the video data according to aspecific codec, and optionally a specific color depth and/or frame size.For example, Encoders 225 may be configured to encode video data toAdobe Flash® standard, .flv, .wav, .avi, .mpg, H.264, H.263, On2, VP6,VC-1, WMA, and/or other codecs discussed herein.

A member of Video DEPs 245 may include one, two or more Encoders 255.These encoders may be configured to encode to different codecs and/orthe different formats of the same codec. For example. Encoder 255A maybe configured to encode according to a Flash standard at a first framesize and color depth while Encoder 255 is configured to encode to thesame Flash standard at a second frame size and color depth. The identityof Encoders 255 within each member of Video DEPs 245 is typicallydetermined at the time the Video DEP 245 is provisioned. For example, asingle command or command set may be used to create (provision) VideoDEP 245A and specify which components the created Video DEP 245A shouldinclude. The creation of Video DEP 245A is discussed further elsewhereherein. A member of Video DEPs 245 including two or more Encoders 255 isalternatively viewed as two or more separate Video DEPs 245 having somebut not all components in common. For example, Video DEP 245A and VideoDEP 245B may have the same Preprocessor 250 but different Encoders 255.

In one example, Encoder 255A is configured to use a codec for H.264while Encoder 255B is configured to use a codec for H.263. Having two ormore different encoders available enables Video DEP 245A to changeencoding during delivery of a video stream. The change in encoding canbe from one type of encoding to another, or merely a change incharacteristics of a specific type of coding. For example, thecharacteristics may change in terms of color depth, number of frames persecond, encoding options, number of pixels, and/or the like. In someembodiments, Video DEP 245A is optionally configured to switch betweenEncoder 255A and 255B in response to a change in the characteristics ofClient 110A or the communication channel between Client 110A and VideoSource 130.

In practice, when a different codec other than that being used isrequired a new Video DEP 245 is spawned and executed in parallel withthe current Video DEP 245 for a short period of time. The new Video DEP245 is optionally a branch of the original Video DEP 245. For example,some components of the original Video DEP 245 may be used in the newVideo DEP 245. These two Video DEP 245 may have components logicallyarranged in a tree structure.

In some embodiments, Video DEP 245 are configured to use two or moredifferent encoders to generate two or more different video streams atthe same time. These video streams are based on the same gameenvironment rendered by Graphics Processing Unit 255 and, thus, includeessentially the same materials (with the possible exception of overlays)and can be sent to different places. For example, one of the videostreams can be sent to Client 110A while the other is sent to Client110B. Alternatively, one of the video streams can be sent to Client 110Aand the other to a website where third parties can watch the video. Thiswebsite is optionally part of a social networking site or a gameplayer's site. The two different video streams may be different in theframe rate, encoding type, frame size, color depth, etc. For example avideo stream delivered to a social networking website can be of muchlower quality than the video stream delivered to a game player who isplaying a game using Client 110A. The second video stream may bedirected to a game player who is playing the game or to people who aremerely observing the game play. A video stream is optionally directed tomore than one place.

Video DEP 245A optionally includes one or more Post Processors (Post)260. Individual examples of Post Processors 260 are labeled 260A and260B. Post Processors 260 are configured to package an encoded videoframe in a container in a format appropriate for communication over anetwork according to a public or proprietary network protocol. Forexample, some protocols such as Adobe RTMP require post processing whileother video standards such as H.264 Annex B do not require postprocessing. Each of Post Processors 260 may be associated with aspecific member of Encoders 255, or several Post Processors 260 may beconfigured to receive encoded video frames from one member of Encoders255.

The output of Post Processors 260 is directed to I/O Device 145 fordelivery to one or more of Clients 110. The Elements of Video DEPs 245discussed herein include hardware, firmware and/or software stored on acomputer readable medium. For example, each of Video DEPs 245 mayrepresent a set of software loaded into memory and executing using anelectronic processor.

Audio DEPs 270 are configured to receive audio data generated by membersof Game Logic 210 and encode this audio data to audio packets. Theencoded data packets are then sent to Clients 110 using I/O Device 145and Network 115. Audio DEPs 270 are also optionally configured to placean audio overlay on the audio data received from Game Logic 210.Different Audio DEPs 270 are individually labeled 270A, 270B, 270C, etc.

As with Video DEPs 245, Audio DEPs 270 are typically allocated as neededand assigned to process audio data from a specific member of Game Logic210. This audio data may be based on events within a video game. Forexample, actions taken by one or more players of the video game mayresult in audio data according to game rules (e.g., falling in watermakes a splashing sound). Allocation of Audio DEP 270A is typicallyresponsive to the same factors and performed in the same manner asallocation of Video DEP 245A. For example, the elements included inAudio DEP 270A may be responsive to needs of one of Clients 110. AudioDEP 270 may receive audio data from Shared Memory 240 and/or directlyfrom one of Virtual Audio Drivers 285.

Audio DEP 270A optionally includes a Grabber 275 configured to receiveaudio data from one of Virtual Audio Drivers 285, to receive audio datafrom one or more sources external to Video Server System 120, and/or toget audio data from Shared Memory 240. For example, in some embodimentsGrabber 275 is configured to retrieve data received from sourcesexternal to Video Source 130, such as one or more of Clients 110, overthe internet. The audio received, from any of these sources, can includevoice audio, music, sound effects, sound tracks, alerts, and/or thelike. For example, the voice of a game player using Client 110A may bereceived from Client 110A, or the game player may provide music to beheard by a game player using another of Clients 110. Audio data may bereceived, as audio streams, from more than one game player in parallel.

Audio DEP 270A optionally further includes Overlay Logic 295. OverlayLogic 295 is configured to place one or more audio overlays on audiodata generated by a member of Game Logic 210. The audio overlays caninclude audio, music, sound effects, sound tracks, alerts, and/or thelike. For example, in some embodiments, an audio overlay includes voicedata received as data packets from one or more game players over theinternet. These data packets are optionally decoded by Overlay Logic 295prior to being included in the overlay. Placement of overlays on theaudio data generated by a member of Game Logic 210 optionally includesaddition of this data to the generated audio data. All or part of anaudio overlay can be received from a source external to Audio Source130. For example, in some embodiments the overlay includes audio datareceived at I/O Device 145 via the internet and Overlay Logic 290includes audio processing logic configured to generate an audio overlayfrom this audio data. When referring to audio overlays, the identitiesof which data is the overlay and which is the data on which the overlayis being applied may be arbitrary as, in some embodiments, audio datadoes not have a sense of layering order that can be possible in videodata.

Audio DEP 270A further includes an Audio Encoder 280A. Audio Encoder280A is configured to encode the audio data generated by a member ofGame Logic 210 and the audio overlay into an audio packet according toan audio codec. The encoding is typically performed after the audiooverlay(s) have been applied to the audio data.

Grabber 275, Audio Encoder 280A and Overlay Logic 295 include hardware,firmware, and/or software stored on a computer readable medium. AudioDEP 270B and Audio DEP 270C are alternative embodiments of Audio DEP270A.

The operation of Video Source 130 is typically managed by a Controller265. Controller 265 includes hardware, firmware and/or software storedon a computer readable medium. For example Controller 265 may includesoftware stored in memory and executed using a microprocessor.

In some embodiments, Controller 265 is configured to provision instancesof Game Logic 210A, Virtual Audio Driver 285 and Virtual 3D Video Driver220 in response to a request to play a game. For example, if a requestfor a specific video game is received from Client 110A, Controller 265may retrieve Game Logic 210A, Virtual Audio Driver 285 and Virtual 3DVideo Driver 220 from Storage 155 and place these in working memory.Game Logic 210A may be placed in communication with both Client 110A andGame Server 125. Game Logic 210A is then executed using Processor 150 inorder to play the video game.

In some embodiments, Controller 265 is configured to allocate instancesof Video DEP 245 and/or Virtual Audio Driver 285 in response to arequest to play a game. For example, Controller 265 may be configured tofirst determine or receive the requirements for Video DEP 245A and AudioDEP 270A and then provision computing instructions that satisfy thoserequirements. Video DEP 245A and Audio DEP 270A are then associated withone of Game Logic 210 and one of Virtual 3D Driver 220. The provisionedVideo DEP 245A is optionally configured to retrieve raw video data froma specific area within Shared Memory 240.

In some embodiments the requirements for Video DEP 245A and/or Audio DEP270A are determined by querying a user of Client 110A. For example,Controller 265 may be configured to cause text messages to appear onClient 110A, the messages requesting that a user of Client 110A entercharacteristics of a video stream desired by a user. Thesecharacteristics can include a connection type, a frame size, an encodingscheme, a frame rate, a color depth, and/or the like. The enteredcharacteristics are conveyed back to Controller 265, which then usedthese characteristics to select elements to include in Video DEP 245Aand/or Audio DEP 270A when they are provisioned. For example, if theuser requests a specific encoding scheme, then an Encoder 255 configuredto use the requested encoding scheme is selected and included in VideoDEP 245A.

In some embodiments the requirements for Video DEP 245A and/or Audio DEP270A are determined automatically. For example, information gatheredusing a client qualifier can be used to determine the requirements forVideo DEP 245A. In this example, if Client 110A includes only aMicrosoft Silverlight® decoder and an Adobe Flash® decoder, then atleast one of the corresponding encoder configurations or variantsthereof is required in Video DEP 245A. Characteristics of the videostream to be generated using Video DEP 245A, such as frame size, colordepth and frame rate, can also be determined by Controller 265 in thisway.

The determination of requirements for Video DEP 245A and/or Audio DEP270 may include both the use of characteristics provided by a user andcharacteristics automatically determined. In some embodiments, thepossible characteristics are limited by a user's account type, paymentand/or subscription. For example, a user may pay to receive a higherquality video stream.

Controller 265 may also manage sharing of Shared Memory 240. Forexample, Controller 265 may configure Video DEP 245A and/or Audio DEP285A to use specific memory locations within Shared Memory 240 and/orController 265 may be configured to configured Video DEP 245A and/orAudio DEP 285A to access Shared Memory 240 in response to certain timingsignals. The timing signals being timed for synchronization of access toShared Memory 240 by different Video DEPs 245.

FIG. 3 illustrates further details of Add-on Manager 160, according tovarious embodiments of the invention. In general Add-on Manager 160 isconfigured to enable remote selection and installation of add-ons, tofacilitate execution of add-ons using add-on environments, to manageadd-on data, to associate add-ons and add-on data with specific user(game player) accounts or avatars, and/or the like. Add-on Manager 160typically includes software stored on a computer readable medium, butmay also include firmware or hardware. Add-on Manager 160 can bedisposed on a single computing device or distributed among more than onecomputing device. For example, in various embodiments, components ofAdd-on Manager 160 are disposed on Video Server System 120, Add-onServer 165 and/or Client 110B.

Add-on Manager 160 includes an Add-on Installer 310 configured forinstalling add-ons in association with a computer program. As usedherein, the association of an add-on and computer program is meant toindicate that the add-on is configured to augment or otherwise extendthe functionality of the computer program. Typically, the add-on isconfigured for association with a specific computer program. Theinstallation is optionally performed remotely, e.g., an installation atVideo Server System 120 can be performed in response to commandsreceived from Client 110B via Network 115. These commands are receivedand processed by Add-on Manager 160.

The installation performed by Add-on Installer 310 can include creationof an Add-on Environment 287 having characteristics compatible withexecution of the add-on. As discussed elsewhere herein, an Add-onEnvironment 287 can include a directory structure, computinginstructions configured to redirect communications, etc. For example,the installation can include the creation of directories,subdirectories, or other elements of a directory structure, relative toGame Logic 210. In some embodiments, the installation includes theaddition of computing instructions configured to redirect messages toother devices. The installation of these computing instructions can beat locations of Game Logic 210 and/or of the add-on.

Installation can be commenced and controlled via an Add-on Interface320. Add-on Interface 320 includes a user interface configured forpresentation to users of Clients 110, computing instructions configuredto generate and communicate this user interface via Network 115, and/orcomputing instructions configured to receive commands entered by theuser via the interface. In various embodiments, the user interface isconfigured for a user to select add-ons to be installed, pay for add-onor add-on data storage space, select which add-ons should be activatedfor particular avatars, turn on and off add-ons, update add-ons, tomanaged an approved add-on list, and/or de-install add-ons. Selectionsand the entry of commands is typically accomplished by clicking orentering text, etc., in that part of Add-on Interface 320 that ispresented by a user. The results of these actions are communicated toother parts of Add-on Interface 320 for processing. Part of Add-onInterface 320 is optionally included in a video stream communicated toClients 110 as an overlay.

Using Add-on Interface 320 a user can install one or more add-onsremotely on Video Server System 120 or Add-on Server 165, and/or installappropriate computing instructions for redirection at locations ofadd-ons and Game Logic 210 remote from Clients 110. For example, a usermay select one or more add-ons from a list of approved add-ons. Thislist of approved add-ons is optionally stored in Storage 155 or Add-onServer 165. Approved add-ons are add-ons that have been qualified foruse with Game Logic 210 by a game publisher, a manager of Video ServerSystem 120, or other authority. Using Add-on Interface 320 the user canoptionally also select which account, avatar and/or game the add-onsshould be associated with and where the add-on should be stored, e.g.,at one of Clients 110, at Video Server System 120, and/or at Add-onServer 165. For example, the user may specify that an add-on should onlybe used with one or more specific avatar but not with another avatar.

In some embodiments of the invention a user is limited in the number ofadd-ons and/or amount of add-on data storage the user has available.This limit is managed by Add-on Manager 160 and may be increased bypurchasing additional space. For example, in one embodiment, a userreceives three virtual “slots” in which add-ons can be placed and canpurchase additional slots by paying money or performing some action suchas bringing in additional players.

Add-on Manager 160 is optionally configured for a user to turn on andoff installed add-ons. For example, a user may wish to use an add-ononly when engaging in certain in-game activities such as player versusplayer encounters. Turning off an add-on does not necessarily mean thatit is de-installed. In some embodiments, add-ons that are off do notrequire an add-on slot. In these embodiments, a user can have moreadd-ons installed than available slots. Alternatively, each installedadd-on, whether or not active, may require a slot.

In some embodiments Add-on Manager 160 is configured to update add-ons.These updates can be manual and/or automatic. Manual updates include,for example, a user providing Add-on Manager 160 with a web address ofan add-on publisher or with add-on code. The add-on can be retrieved bythe Add-on Manager 160 from the publisher or from one of Clients 110.Automatic add-on update includes automatic retrieval of an add-on froman add-on publisher or other source. For example, when an add-on isdetected as being out of date, Add-on Manager 160 is optionallyconfigured to automatically request an update from a publisher or othersource of that add-on and automatically install the update if available.This process can be transparent to the game player. In some instances,Add-on Manager 160 will wait until an add-on is available from thesource and then perform the update.

With either manual or automatic add-on updates, Add-on Manager 160 isoptionally automatically configured to detect when an add-on is out ofdate and notify the user. Add-on Manager 160 may also be configured toreceive a notice from an add-on publisher that a new version of anadd-on is available.

Add-on Manager 160 is optionally configured for a user of Clients 110 tomanage add-ons via Add-on Interface 320. This management performed viaAdd-on Interface 320 can include, for example, updating add-ons,purchasing add-ons, configuring add-ons, moving add-ons, viewinginstalled add-ons, activating and inactivating add-ons, installingadd-ons, de-installing add-ons, and/or the like. In some embodiments,Add-on Interface 320 is configured to perform any add-on managementfunction that could be performed on a client side add-on. Thede-installation optionally includes the removal of computinginstructions configured for the redirection of messages between anadd-on and Game Logic 210, and/or the removal of an Add-on Environment287. As discussed elsewhere herein, in some embodiments Add-on Manager160 is configured to automatically update add-ons. The update mode isoptionally managed using Add-on Interface 320.

Add-on Manager 160 optionally further includes an Account Manager 325configured to manage the association of add-ons and/or add-on data witha user of Client 110B or other Clients 110. When an add-on is installedby a user (game player), Account Manager 325 associates the add-on witha specific video game, user, account of the user, and/or avatar.Typically this association is stored as part of a list of add-onsinstalled by each user. When a user starts a video game, Account Manager325 reads this list and identifies add-ons associated with the user.These add-ons and the appropriate add-on environments are thenprovisioned along with the provisioning of Game Logic 210, Video DEP245A and/or Audio DEP 270A. Add-ons are optionally further provisionedor de-provisioned as the user switches between avatars or activates andde-actives add-ons.

Account Manager 325 manages storage of add-on data in association with auser when the user is not playing a video game. The add-on data may bestored at Storage 155, Add-on Server 165 and/or Client 110. When gameplay is started this data is typically copied to the appropriate Add-onEnvironment 287 for use by one or more provisioned add-on. When gameplay is completed the copy of the add-on data in the Add-on Environment287 is used to update the stored copy. The add-on data is stored in adatabase, file structure or the like and includes identifyinginformation, such as a record index or a file name, configured toidentify the add-on data as belonging to a specific user (i.e., gameplayer). In some embodiments Account Manager 325 maintains an indexconfigured to indicate which files of add-on data belong to a specificuser and/or are associated with a specific account or avatar.

In various embodiments, Account Manager 325 is configured to manage thenumber of add-on slots and/or storage accessible to each user, to chargethe user for extra slots and/or storage, and/or to offer the user extraslots or storage for purchase. For example, if a user attempts toactivate more add-ons than there are slots available, Account Managermay use Add-on Interface 320 to offer the user an opportunity topurchase more slots.

Add-on Manager 160 optionally further includes an Add-on Updater 330,configured to update add-ons. Updates can occur when a new version of anadd-on is available or in response to new versions of Game Logic 210.For example, when a new version of Game Logic 210 is released by a gamepublisher third party add-ons for that game will typically need to betested and/or updated. Add-on Updater 330 is configured to monitor forsuch events, e.g., release of new versions of add-ons and Game Logic210, and to perform updates in response. In some embodiments, theprocess includes receiving information that indicates that a new versionof Game Logic 210 is available, notifying add-on publishers that anupdated version of an add-on is required, receiving the updated add-on,and replacing the out of date add-on with the updated add-on. Any or allof these steps can be performed automatically or can involve input froma user of Client 110B. In some embodiments, Add-on Updater 330 isconfigured to automatically retrieve add-ons from a third partylocation, such as an ftp server or website.

Add-on Manager 160 further includes an Add-on Storage 335. Add-onStorage 335 includes at least one electronic storage device configuredto store data, images, and/or executable computing instructions. Forexample, Add-on Storage 335 can include volatile or non-volatile memory,a hard drive, and optional drive, static or dynamic random accessmemory, and/or the like. Add-on Storage 335 is optionally distributedamong a plurality of devices including Video Server System 120, Add-onServer 165 and Clients 110.

Add-on Storage 335 optionally includes Add-ons 340. Add-ons 340 comprisedata, images, executable code, scripts, or other game content configuredto be used to extend the functionality of Game Logic 210. Add-ons 340may be installed as add-ons on Video Server System 120, Add-on Server165, Clients 110, or elsewhere. Add-ons 340 are optional in embodimentswhere add-ons are retrieved for execution directly from third partysources. Add-on Storage 355 optionally further includes an Approval List345 of approved Add-ons 340. The computing instructions within Add-ons340 are optionally shared by more than one game player. For example, asingle add-on, when provisioned, can be associated with both Game Logic210A and Game Logic 210B, different computing threads independentlyaccessing the computing instructions.

Add-on Storage 355 optionally includes API Filters 350. API Filters 350comprise the computing instructions configured to redirectcommunications between one of Add-ons 340 and Game Logic 210, asdiscussed elsewhere herein. Such computing instructions may beconfigured such that an add-on on Add-on Server 165 or one of Clients110 can be used to extend functionality of Game Logic 210 on VideoServer System 120. Alternatively, using API Filter 350 an add-on onVideo Server System 120 or on Add-on Server 165 may be used to extendthe functionality of Game Logic 210 installed on one of Clients 110.When API Filters 350 are installed in proper locations, an add-on can beused to extend functionality of a computer program even when the add-onand computer program are separated by Network 115.

Add-on Storage 355 optionally includes Add-on Data 355 included with,generated or organized by an add-on. Add-on Data 355 may include datagenerated by more than one add-on. Typically, this data is copied to theappropriate Add-on Environment 287, when game play is initiated or anAdd-on 340 is activated. When game play is completed the add-on data inthe Add-on Environment 287 is used to update the copy of the add-on datastored in Add-on Storage 355.

FIG. 4 illustrates further details of a client, according to variousembodiments of the invention. I/O 405 includes an input/output deviceconfigured for Client 110 to send and/or receive information such asvideo, commands, add-ons, add-on data, requests for information, a gamestate, client identities, player identities, game commands, securityinformation, audio data, and/or the like. In some embodiments, I/Oincludes a modem or Ethernet port.

Client-side Storage 410 includes non-transitory analog and/or digitalstorage devices. For example, Client-side Storage 410 may include ananalog storage device configured to store video frames. Client-sideStorage 410 may include a computer readable digital storage, e.g. a harddrive, an optical drive, or solid state storage. Client-side Storage 410is configured (e.g. by way of an appropriate data structure or filesystem) to store video frames, artificial frames, a video streamincluding both video frames and artificial frames, audio frame, an audiostream, add-ons, add-on data, and/or the like. Some of these add-ons canbe associated with the instance of Game Logic 210 on Client 110B, whilethe same Game Logic 210 is also associated with add-ons installed on adevice geographically remote form Client 110B.

API Filter 350 is part of an Add-on Environment 287 configured toredirect communications as discussed elsewhere herein. Processor 420includes both a central processing unit (CPU) and at least one graphicsprocessing unit (GPU). Display 430 is configured to display a videostream generated local to Client 110B and/or generated by Video ServerSystem 120.

User Input 440 includes, for example, a keyboard, a joystick, a pointingdevice, a telephone, a force feedback device, a motion and/or locationsensing device, a mouse, a touch screen, a neural interface, a camera,input devices yet to be developed, and/or the like.

FIG. 5 illustrates methods of installing and/or using an add-on,according to various embodiments of the invention. These methods may beapplied in embodiments in which the add-on is installed on Add-on Server165, Video Server System 120, or one of Clients 110, and embodiments inwhich Game Logic 210 is installed on one of Clients 110 or Video ServerSystem 120. Common to all these embodiments is that at least one of anadd-on and the associated Game Logic 210 are installed and used at alocation other than one of Clients 110 and separated from the one ofClients 110 by Network 115. For example, if Game Logic 210 is installedon Client 110B then at least one associated add-on is installed and usedon a different member of Clients 110, Add-on Server 165, Video ServerSystem 120, or a third party location. If Game Logic 210 is installed ina location other than Client 110B then add-ons can be installed onClient 110B, or any of the other locations discussed herein. Those stepsof FIG. 5 involving installation of add-ons are optionally performedseparately from those steps involving using the installed add-ons. Bothinstallation and execution of an add-on may be initiated by a gameplayer using one of Clients 110, or may be specified as part of adefault configuration for Game Logic 210. Some embodiments includeinstalling more than one add-on.

Considering first those embodiments in which Game Logic 210 is installedon Video Server System 120, the installation process can begin with aSelect Add-on Step 510. In this step the add-on to be installed isselected, typically by a game player using one of Clients 110. Theselection can be from a list of add-ons presented to the game playerusing Add-on Interface 320, or may include the game player selecting anadd-on from a storage location or a third party source.

In some embodiments only approved add-ons can be selected. In theseembodiments a Confirm Add-on Step 515 is optionally used to confirm thatthe selected Add-on 340 is included in an approved list of add-ons. Thischeck can include looking for a name (or other identifier) of theadd-ons in the approved list or comparing a checksum, URL, or othersignature of an approved add-on with a corresponding characteristic ofthe selected add-on.

In an Establish Environment Step 520 Add-on Manager 160 is used toestablish an instance of Add-on Environment 287 for executing theselected add-on. In various embodiments, the Add-on Environment 287 isestablished on Video Server System 120, Add-on Server 165, or one ofClients 110. For example, on Video Source 130 establishing the Add-onEnvironment 287 may include creating an appropriate directory structuresuch that the add-on can be installed in a location expected by theassociated Game Logic 210. In a more specific example, if Game Logic 210is configured to look in a subdirectory called “add-ons,” relative to adirectory of Game Logic 210, then Establish Environment Step 520 mayinclude creating this subdirectory in the proper location.

Establishing Add-on Environment 287 on Add-on Server 165 or one ofClients 110 typically includes placing computing instructions configuredto redirect communications between Game Logic 210 and the add-on, atappropriate locations. For example, a first API Filter 350 may beinstalled at an API of Game Logic 210 and a second API Filter 350 may beinstalled at a location at which the add-on will be executed. These APIFilters 350 are configured to redirect messages to each other such thatthe add-on and the associated Game Logic 210, which are configured tooperate together on the same computing device, can instead operate bycommunicating with each other over Network 115. Establishing Add-onEnvironment 287 optionally includes configuring each of the API Filters350 such that can communicate with each other, regardless of where theyare installed. For example, each of API Filters 350 may be configuredwith an IP address or universal resource locator of the correspondingAPI Filter 350.

In a Receive Add-on Step 525 the add-on to be installed is received fromStorage 155, Add-on Storage 335, one of Clients 110 or a third partylocation. The add-on is optionally received via Network 115. The add-onis received at the location it is to be executed, e.g. the Add-onEnvironment 287 established in Establish Environment Step 520. Someembodiments of Receive Add-on Step 525 include communicating the add-onfrom, Add-on Storage 335, Client 110B or a third party location to VideoServer System 120. Some embodiments of Receive Add-on Step 525 includecommunicating the add-on from Add-on Storage 335, Storage 155, or athird party location to Client 110B.

In a Place Step 530 the received add-on is installed/placed in theAdd-on Environment 287 established in Establish Environment Step 520.This installation can occur within a directory structure of Video Source130, or at the locations of API Filters 350 installed on Add-on Server165, video Source 130, and/or Clients 110. Installation can includeprovisioning of computing instructions of the add-on and optionallyprovisioning of default add-on data. Establish Environment Step 520 isoptionally performed after or at the same time as Place Step 530. Forexample, API Filters 350 can be installed after the add-on is installed.Place Step 520 optionally includes provisioning of add-on data alongwith the add-on. The add-on data may or may not be from the samelocation as the add-on. For example, the add-on may be received fromAdd-on Server 165 while the add-on data is received from Client 110B.The received add-on data can be default add-on data or add-on data thathas been modified by actions of a game player.

In an optional Associate Step 535 the add-on installed in Place Step530, and optionally the provisioned add-on data, are associated with aone of Clients 110 or a user of one of Clients 110, i.e. a game player.This association can involve inclusion of an identifier of the add-on ina list of add-ons assigned to the user or copying of the add-on into adirectory assigned to the user. The association with the user can bespecific to an account of the user, a user, a particular video game, oneor more particular avatars, and/or the like. Associate Step 530 may beoptional when the add-on is installed on one of Clients 110 or in alocation that is already associated with an account, a user, etc. Insuch cases the association with a location, device, address, etc. mayserve in place of or in addition to an association with a particularuser.

Once an add-on is associated with an account, a user, or location, etc.it can be provisioned and/or used every time the user requests executionof Game Logic 210. For example, one or more add-ons associated with auser can be provisioned at approximately the same time as Game Logic210A and Video DEP 245A. The provisioning process can include variationof Steps 510 through 530 as described herein. For example, when anadd-on is provisioned after previously being installed Select Add-onStep 510 can merely include parsing a list of add-ons previouslyassociated with a user or location, etc. Receive Add-on Step 525 canmerely include retrieving the add-on and associated add-on data from oneor more storage locations such as Storage 155, Add-on Storage 335, orClient Side Storage 410.

Once an add-on and any associated add-on data are provisioned, use ofthe add-on can begin with an Execute Game Step 540. In this step GameLogic 210 is executed, typically in response to a request received fromone of Clients 110. The execution optionally includes communicationsbetween Video Server System 120 and Game Server 125 such that a state ofa multiplayer video game is maintained at Video Server System 120.

In a Receive Command Step 545 a game command is received by one of GameLogic 210 from one of Clients 110. This typically occurs via Network115. The game command can be a command configured for control of or useby an add-on. For example, the game command can be configured to causethe add-on to display certain information, to perform certain functionsor can include data to be processed by the add-on.

In a Forward Command Step 550 the game command received in ReceiveCommand Step 550 is identified as a command for the add-on and forwardedto the add-on. In embodiments in which the add-on is installed on thesame computing device as Game Logic 210 that received the command, thecommand can be passed directly from the API of this Game Logic 210 to anAPI of the add-on. For example, the game command can be passed by theGame Logic 210A API to a memory location of the Add-on API.

In embodiments in which the add-on is installed on a different computingdevice than the executed Game Logic 210, API Filters 350 are used tocommunicate the received command between the Game Logic 210 and theadd-on. For example, the command may be first communicated from GameLogic 210A to a first API Filter 350 on Video Source 130. This first APIFilter 350 is configured to appear as the Add-on API from the point ofview of the Game Logic 210A API. Thus the communication from Game Logic210A to the first API Filter 350 can be accomplished by providing thecommand, or a pointer to the command, to a memory location of the firstAPI Filter 350. The first API filter 350 is configured to communicatethe command to a second API Filter 350 at the location of the add-on,e.g., at one of Clients 1108 or Add-on Server 165. This communicationcan occur over Network 115 using TCP/IP, UDP/IP, or some othercommunication standard. Upon receipt of the command, the second APIFilter 350 is configured to forward the command to the add-on. Thesecond API Filter 350 is configured to appear as the Game Logic API fromthe point of view of the add-on. Thus, communication from the second APIFilter 350 to the add-on can be accomplished by providing the command,or a pointer to the command, to a memory location of the add-on.

In a Process Command Step 555 the command is processed using the add-on.Processing can include the generation of add-on data. For example, thecommand may include data provided by a user or by Game Logic 210A. Thecommand may cause the add-on to record or detect events in the game andgenerate add-on data based on these events. The command may cause theadd-on to provide audio or image data for inclusion in an audio or videostream generated by Video Server System 120 to one of Clients 210. Thecommand may cause an image to be converted between formats. For example,be converted to a pdf file. This audio or image data, or any other dataresulting from Process Command Step 555, is optionally communicated toVideo Server System 120 or one of Clients 110, for example, via Network115. The command may cause any other result known in currently existingadd-on technology.

In an Optional Store Step 560 add-on data generated in Process CommandStep 550 is stored. This storage may be proximate to the add-on, oralternatively at a location in communication with the add-on via Network115. For example, regardless of where the add-on is located the data maybe stored on Video Server System 120, Add-on Server 165, and/or one ofClients 110. Storage may occur in real-time or after a delay. Forexample, in some embodiments a working copy of the add-on data local tothe add-on is used during a specific execution of Game Logic 210A, andwhen this execution is completed the add-on data is stored elsewhere.

In an optional Generate Step 565 video data is generated as a result ofthe add-on processing the game command. For example, the game commandmay represent a request to have add-on data, locations, maps, text,notices, items, and/or the like displayed to a game player (or a soundpresented to the game player). The video data may include twodimensional or three dimensional images. For example, to indicate alocation of a resource a two dimensional image/symbol may be read fromstorage and converted to video data by being applied as an overlay. Theuse of overlays is discussed further in U.S. patent application Ser. No.12/826,130 filed Jun. 29, 2010 and entitled “Video Game Overlay.” Thedisclosure of this patent application is hereby incorporated herein byreference. Alternatively, the video data may be the result of the add-onadding a two or three dimensional object to a game environment which isthen rendered to video data using GPU 235.

In an optional Provide Step 570, the video data generated in GenerateStep 560 is provided to one or more of Clients 110 as part of a videostream. Typically, these Clients 110 include the member of Clients 110from which the game command was received in Receive Command Step 545,and the video stream is provided via Network 115. For example, if thegame command is received from Client 110B the video may be provided toClient 110A and a third destination, as well as Client 110B. The thirddestination can be a storage place, a website, a social networking site,a television, a phone, and/or the like.

Now considering those embodiments in which Game Logic 210 is installedon Client 110B and at least one add-on is external to Client 110B duringuse. In these embodiments Video Server System 120 is optional as thegame environment can be generated based on a game state, the gameenvironment can be rendered to video, and the video can be displayed,all on Client 110B. The game state can be based on state informationreceived from Game Server 125, received from another member of Clients110, and/or based solely on information within Client 110B. The stepsillustrated in FIG. 5 may be varied as follows:

In Establish Environment Step 520 the Game Logic 210 and the add-on arenecessarily on different computing devices. This is because, in theembodiments being discussed, Game Logic 210 is on Client 110B and atleast one of Game Logic 210 and an add-on are on different computingdevices separated by Network 115. Establish Environment step 520,therefore includes provisioning API Filters 350 at both the location ofGame Logic 210 and the location of the add-on. In some embodiments, oneGame Logic 210 and one add-on are both on Client 110B while at least oneother add-on is on a different computing device, e.g. Add-on Server 165and/or Video Server System 120. In these embodiments, EstablishEnvironment step 520 is directed toward creating an Add-on Environment287 for the add-on(s) on the different computing devices.

In Receive Add-on Step 525 the add-on is received at Add-on Server 165,Video Server system 120, a different Client 110, or a third partylocation, such as a web server, file server, or the like controlled by adeveloper of the add-on. In Place Step 530 the add-on is installed at alocation other than the Client 110 on which the instance of Game Logic210 to be augmented by the add-on is installed. For example, if GameLogic 210 is disposed on Client 210B, the add-on can be installed onClient 210C, Add-on Sever 165, Video Server System 120, or a third partylocation. Place Step 530 optionally includes communicating add-on datafrom Client 210B, Add-on Server 165, or Video Server System 120 to thelocation at which the add-on is installed.

In Associate Add-on Step 535 the list of add-ons used to save add-onassociations is optionally stored on Client 210B. Execute Game Step 540and Receive Command Step 545 are optionally confined to Client 110B. Forexample, Receive Command Step 545 may include communicating a commandfrom User Input 440 to Processor 420, rather than communicating thecommand via Network 115. In Forward Command Step 550 the commandreceived in Receive Command Step 544 is forwarded from Client 110B tothe location of the add-on. Any data resulting from Process Step 555 isoptionally communicated back to Client 210B.

In optional Store Step 560 data generated in Process Command Step 550 isstored on Client 110B, the location of the add-on, and/or elsewhere.Provide Step 570 can include merely providing the video data generatedin Generate Step 560 from Processor 420 to Display 430. Although, issome embodiments, this video data can be also provided to other membersof Clients 110 or to third party locations discussed herein.

Returning now to the more general case in which Game Logic 210 can bedisposed on either Clients 110 or Video Server System 120. FIG. 6illustrates methods of automatically updating an add-on, according tovarious embodiments of the invention. The add-on is typically updated inresponse to receipt of an updated version of Game Logic 210. Anycombination of the steps illustrated in FIG. 6 can be performedautomatically. It is possible that a request for an updated add-on ismade before a request is received to execute the updated version of GameLogic 210. The methods illustrated in FIG. 6 are optionally used toupdate more than one add-on in response to receiving one updated GameLogic 210.

In a Receive Game Logic Update Step 610 a new version of Game Logic 210is received at Video Source 130 or Client 210B. This new version istypically received from a publisher of the associated video game or anintermediary thereof. The new version is optionally received via Network115. The new version of Game Logic 210 is configured to replace aprevious version. Receive Game Logic Update Step 610 optionally includesdisabling any out of date add-ons.

In a Determine Add-on Usage Step 615 it is determined whether an add-onhas been used to augment the functionality of the previous version ofGame Logic 210. Determine Add-on Usage Step 615 can include, forexample, examining a list of add-ons associated with specific Game Logic210, video game, users, user accounts, Clients 110, etc. This list maybe maintained by Account Manager 325. Alternatively, Determine Add-onUsage Step 615 can include examining storage locations configured tostore add-ons and looking for add-ons configured to augment Game Logic210.

In a Send Request Step 620 a request for an updated add-on made. Thisrequest is typically made to a third party, such as a publisher of theadd-on or intermediary thereof. The request includes an address to whichthe update should be provided, and optionally an identity of the add-onand new version of Game Logic 210 with which the add-on should becompatible. For example, the request may indicate that a new version of“Deadly Boss Mods” for the “World of Warcraft version 6.542” Game Logic210 be sent to an address of Video Server 120, Add-on Server 165 orClient 110B. The request is optionally sent via Network 115. SendRequest Step 620 can be performed between a time the updated version ofGame Logic 210 is received in Receive Game Logic Update Step 610 and afirst time a request to execute the updated version of Game Logic 210 isreceived. Send Request Step 620 can be performed automatically inresponse to receiving the new version of Game Logic 210 in Receive GameLogic Update Step 610. Send Request Step 620 is optionally repeateduntil an updated add-on is received. For example, the request caninitially be sent every 10 minutes and later sent every hour.

In a Receive Add-on Update Step 625 the updated add-on is received atthe address provided in Send Request Step 620. The updated add-on isconfigured to augment the new version of Game Logic 210 received inReceive Game Logic Update Step 610 and can be received via Network 115.Receive Add-on Update Step 625 is optionally performed between the timethe updated version of Game Logic 210 is received in Receive Game LogicUpdate Step 610 and a time a first request to execute the updatedversion of Game Logic 210 is received.

In an Install Step 630 the updated add-on received in Receive Add-onUpdate Step 625 is installed. This installation may be in an add-on onstorage such as Add-on Storage 335 or Client-side Storage 410.Alternatively, the installation may be in an Add-on Environment 287where it can be used without further provisioning. The installationprocess is optionally similar to Place Step 530 as discussed withrespect to FIG. 5 . Install Step 630 is optionally performedautomatically, e.g., without need for a game player to order theinstallation.

In a Receive Execution Request Step 635 a request to execute the newversion of Game Logic 210, received in Receive Game Logic Updates Step610, is received. This request is received from a game player via one ofClients 110, and is optionally received at Video Source 130 via Network115. Receive Execution Request Step 635 may occur before or after any ofSend Request Step 620, Receive Add-on Update Step 625 and Install Step630.

In an optional Request Donation Step 640 a request to donate to apublisher of the add-on is presented to a game player. Typically this isthe game player that made the request for execution in Receive ExecutionRequest Step 635. The request to donate is optionally presented as anoverlay on the video stream generated using Video Server System 120.

In an optional Notify User Step 645 the game player that made therequest for execution in Receive Execution Request Step 635 is notifiedthat an new version of the add-on has been received. The notice isoptionally presented as an overlay on the video stream generated usingVideo Server System 120.

Several embodiments are specifically illustrated and/or describedherein. However, it will be appreciated that modifications andvariations are covered by the above teachings and within the scope ofthe appended claims without departing from the spirit and intended scopethereof. For example, while the example presented herein are directed atcomputer games, in other embodiments the systems and methods taught areapplied to other types of computer programs. These other types ofcomputer programs can include graphics programs, spreadsheets, wordprocessors, database programs, accounting programs, project managementprograms, video editing programs, image editing programs, websitedevelopment programs, inventory programs, e-mail programs, filemanagement programs, document management/viewing programs,drawing/computer aided design programs, presentation programs, and/orthe like. Further, while many of the examples discussed herein refer toa single add-on. The systems and methods disclosed can be applied togroups of add-ons. These groups can be handled as an add-on package.

The embodiments discussed herein are illustrative of the presentinvention. As these embodiments of the present invention are describedwith reference to illustrations, various modifications or adaptations ofthe methods and or specific structures described may become apparent tothose skilled in the art. All such modifications, adaptations, orvariations that rely upon the teachings of the present invention, andthrough which these teachings have advanced the art, are considered tobe within the spirit and scope of the present invention. Hence, thesedescriptions and drawings should not be considered in a limiting sense,as it is understood that the present invention is in no way limited toonly the embodiments illustrated.

Computing systems referred to herein can comprise an integrated circuit,a microprocessor, a personal computer, a server, a distributed computingsystem, a communication device, a network device, or the like, andvarious combinations of the same. A computing system may also comprisevolatile and/or non-volatile memory such as random access memory (RAM),dynamic random access memory (DRAM), static random access memory (SRAM),magnetic media, optical media, nano-media, a hard drive, a compact disk,a digital versatile disc (DVD), and/or other devices configured forstoring analog or digital information, such as in a database. Thevarious examples of logic noted above can comprise hardware, firmware,or software stored on a computer-readable medium, or combinationsthereof. A computer-readable medium, as used herein, expressly excludespaper. Computer-implemented steps of the methods noted herein cancomprise a set of instructions stored on a computer-readable medium thatwhen executed cause the computing system to perform the steps. Acomputing system programmed to perform particular functions pursuant toinstructions from program software is a special purpose computing systemfor performing those particular functions. Data that is manipulated by aspecial purpose computing system while performing those particularfunctions is at least electronically saved in buffers of the computingsystem, physically changing the special purpose computing system fromone state to the next with each change to the stored data.

What is claimed is:
 1. A method, comprising: executing at a server gamelogic of a video game to generate video data including one or morerendered video frames, wherein the game logic is executed responsive toa plurality of input commands received over a network from a clientdevice of a player; sending over the network at least one command fromthe server to the client device for execution by an add-on program,wherein execution of the at least one command generates add-on data;receiving over the network the add-on data at the server from the clientdevice; overlaying the add-on data at the server to the one or morerendered video frames; encoding at the server the one or more renderedvideo frames overlaid with the add-on data to produce an encoded videostream; and streaming the encoded video stream over the network from theserver to the client device.
 2. The method of claim 1, wherein theadd-on data is video data.
 3. The method of claim 1, further comprising:executing the add-on program to enable communications with anotherplayer, wherein the add-on data is audio data.
 4. The method of claim 1,further comprising: determining by the game logic that the at least onecommand is directed to the add-on program, wherein the at least onecommand is taken from the plurality of input commands; and wherein thesending over the network includes redirecting the at least one commandto the client device for execution by the add-on program.
 5. The methodof claim 1, further comprising: establishing an instance of an add-onenvironment on the game logic; and creating a file directory for theadd-on program in a directory structure of the game logic, wherein theinstance of the add-on environment is configured to redirect the atleast one command to the add-on program located at the client devicebased on the file directory for the add-on program, wherein the instanceof the add-on environment emulates the add-on program at the game logic.6. The method of claim 1, wherein the add-on program includes a set ofinstructions used to augment the functionality of the video game.
 7. Themethod of claim 1, wherein the add-on program is another video game. 8.A non-transitory computer-readable medium storing a computer program forexecuting a method for executing an add-on program, thecomputer-readable medium comprising: program instructions for executingat a server game logic of a video game to generate video data includingone or more rendered video frames, wherein the game logic is executedresponsive to a plurality of input commands received over a network froma client device of a player; program instructions for sending over thenetwork at least one command from the server to the client device forexecution by an add-on program, wherein execution of the at least onecommand generates add-on data; program instructions for receiving overthe network the add-on data at the server from the client device;program instructions for overlaying the add-on data at the server to theone or more rendered video frames; program instructions for encoding atthe server the one or more rendered video frames overlaid with theadd-on data to produce an encoded video stream; and program instructionsfor streaming the encoded video stream over the network from the serverto the client device.
 9. The non-transitory computer-readable medium ofclaim 8, wherein in the method the add-on data is video data.
 10. Thenon-transitory computer-readable medium of claim 8, further comprising:program instructions for executing the add-on program to enablecommunications with another player, wherein the add-on data is audiodata.
 11. The non-transitory computer-readable medium of claim 8,further comprising: program instructions for determining by the gamelogic that the at least one command is directed to the add-on program,wherein the at least one command is taken from the plurality of inputcommands; and wherein the program instructions for sending over thenetwork includes program instructions for redirecting the at least onecommand to the client device for execution by the add-on program. 12.The non-transitory computer-readable medium of claim 8, furthercomprising: program instructions for establishing an instance of anadd-on environment on the game logic; and program instructions forcreating a file directory for the add-on program in a directorystructure of the game logic, wherein the instance of the add-onenvironment is configured to redirect the at least one command to theadd-on program located at the client device based on the file directoryfor the add-on program, wherein the instance of the add-on environmentemulates the add-on program at the game logic.
 13. The non-transitorycomputer-readable medium of claim 8, wherein in the method the add-onprogram includes a set of instructions used to augment the functionalityof the video game.
 14. The non-transitory computer-readable medium ofclaim 8, wherein in the method the add-on program is another video game.15. A computer system comprising: a processor; and memory coupled to theprocessor and having stored therein instructions that, if executed bythe computer system, cause the computer system to execute a method forexecuting an add-on program comprising: executing at a server game logicof a video game to generate video data including one or more renderedvideo frames, wherein the game logic is executed responsive to aplurality of input commands received over a network from a client deviceof a player; sending over the network at least one command from theserver to the client device for execution by an add-on program, whereinexecution of the at least one command generates add-on data; receivingover the network the add-on data at the server from the client device;overlaying the add-on data at the server to the one or more renderedvideo frames; encoding at the server the one or more rendered videoframes overlaid with the add-on data to produce an encoded video stream;and streaming the encoded video stream over the network from the serverto the client device.
 16. The computer system of claim 15, wherein inthe method the add-on data is video data.
 17. The computer system ofclaim 15, the method further includes: executing the add-on program toenable communications with another player, wherein the add-on data isaudio data.
 18. The computer system of claim 15, wherein the methodfurther includes: determining by the game logic that the at least onecommand is directed to the add-on program, wherein the at least onecommand is taken from the plurality of input commands; and wherein thesending over the network includes redirecting the at least one commandto the client device for execution by the add-on program.
 19. Thecomputer system of claim 15, the method further including: establishingan instance of an add-on environment on the game logic; creating a filedirectory for the add-on program in a directory structure of the gamelogic, wherein the instance of the add-on environment is configured toredirect the at least one command to the add-on program located at theclient device based on the file directory for the add-on program,wherein the instance of the add-on environment emulates the add-onprogram at the game logic.
 20. The computer system of claim 15, whereinin the method the add-on program is another video game.